Improved Architectures and Training Algorithms for Deep Operator Networks

Operator learning techniques have recently emerged as a powerful tool for maps between infinite-dimensional Banach spaces. Trained under appropriate constraints, they can also be effective in the solution operator of partial differential equations (PDEs) an entirely self-supervised manner. In this work we analyze training dynamics deep networks (DeepONets) through lens Neural Tangent Kernel theory, and reveal bias that favors approximation functions with larger magnitudes. To correct propose to adaptively re-weight importance each example, demonstrate how procedure effectively balance magnitude back-propagated gradients during via gradient descent. We novel network architecture is more resilient vanishing pathologies. Taken together, our developments provide new insights into DeepONets consistently improve their predictive accuracy by factor 10-50x, demonstrated challenging setting PDE operators absence paired input-output observations. All code data accompanying manuscript will made publicly available at https://github.com/PredictiveIntelligenceLab/ImprovedDeepONets .